Papers with Visual Genome dataset
Visual Question Answering Dataset for Bilingual Image Understanding: A Study of Cross-Lingual Transfer Using Attention Maps (C18-1)
Copied to clipboard
| Challenge: | Existing literature on visual question answering (VQA) focuses on English, but there are no datasets for other languages. |
| Approach: | They propose a cross-lingual method to make use of English annotation to improve Japanese VQA . they use attention maps generated from English questions to improve the task . |
| Outcome: | The proposed method performs better than using a monolingual corpus in Japanese than using monolingual ones. |
Learning Functional Distributional Semantics with Visual Data (2022.acl-long)
Copied to clipboard
| Challenge: | Functional Distributional Semantics models the meaning of a word as a binary classifier rather than a numerical vector. |
| Approach: | They propose a method to train a Functional Distributional Semantics model with grounded visual data. |
| Outcome: | The proposed model outperforms previous work on learning semantics from Visual Genome on four external evaluation datasets. |
RelCLIP: Adapting Language-Image Pretraining for Visual Relationship Detection via Relational Contrastive Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing visual relationship detection models only use numeric ids of relation labels for training, but ignore semantic correlation between labels. |
| Approach: | They propose a visual Relationship prediction framework that transfers natural language knowledge from Contrastive Language-Image Pre-training models to enhance the relationship prediction. |
| Outcome: | The proposed framework improves visual relationship prediction by matching semantic correlations with relation triplets. |
Improving Visual Question Answering by Referring to Generated Paragraph Captions (P19-1)
Copied to clipboard
| Challenge: | Empirical results show that paragraph captions help answer more visual questions . |
| Approach: | They propose a visual and textual question answering model which uses paragraph captions as input . they use cross-attention to extract related information, then consensus to fuse the inputs . |
| Outcome: | Empirical results show that paragraph captions help answer more visual questions . the proposed model significantly improves the baseline model . |
Augmenting Image Question Answering Dataset by Exploiting Image Captions (L18-1)
Copied to clipboard
| Challenge: | Image question answering requires large amounts of human-annotated data to achieve optimal performance. |
| Approach: | They propose a framework to augment training data by generating additional examples from unannotated pairs of an image and captions. |
| Outcome: | The proposed framework augments training data by generating additional examples from unannotated pairs of an image and captions. |
HaVQA: A Dataset for Visual Question Answering and Multimodal Research in Hausa Language (2023.findings-acl)
Copied to clipboard
Shantipriya Parida, Idris Abdulmumin, Shamsuddeen Hassan Muhammad, Aneesh Bose, Guneet Singh Kohli, Ibrahim Said Ahmad, Ketan Kotwal, Sayan Deb Sarkar, Ondřej Bojar, Habeebah Kakudi
| Challenge: | Existing models for visual question answering are limited to the English language. |
| Approach: | They present a multimodal dataset for visual question answering tasks in the Hausa language. |
| Outcome: | The proposed dataset provides 12,044 gold standard English-Hausa parallel sentences that are semantically identical to the corresponding visual information. |
Learning to Relate from Captions and Bounding Boxes (P19-1)
Copied to clipboard
| Challenge: | Existing methods for classifying images without supervision are limited. |
| Approach: | They propose a top-down attention mechanism to align entities in captions to objects in the image and leverage the syntactic structure of captions for alignment. |
| Outcome: | The proposed model achieves a recall@50 of 15% and recall@100 of 25% on the relationships present in the image and predicts relations that are not present in captions. |